Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add Pyodide support #1456

Merged
merged 32 commits into from
May 28, 2024
Merged

feat: add Pyodide support #1456

merged 32 commits into from
May 28, 2024

Conversation

hoodmane
Copy link
Contributor

@hoodmane hoodmane commented Apr 4, 2023

Resolves #1454. Also close #1002.

TODO:

  • Automate Python installation in certain CI environments?

Working:

  • multidict -- (requires this fix to their config)
  • numpy -- builds wheel but hangs 99% of the way through test suite
  • scikitlearn
  • scikit-image -- works on v0.19.3, doesn't work on v0.20 because of Meson
  • mypy -- NonPlatformWheelError
  • ultrajson -- one test failure due to fork
  • ril -- rust compilation errors
  • pydantic -- one test failure due to asyncio.run

@hoodmane
Copy link
Contributor Author

hoodmane commented Apr 5, 2023

@henryiii could you approve the github actions workflows?

@hoodmane hoodmane force-pushed the emscripten branch 2 times, most recently from 38a706b to e6a77ed Compare April 6, 2023 22:30
@hoodmane
Copy link
Contributor Author

hoodmane commented Apr 7, 2023

@henryiii is the windows_x86 failure a flake?

failed to download pip version 20.0.2, pip download exit code 1
The conflict is caused by:
    The user requested pip==20.0.2
    The user requested (constraint) pip==20.0.2
To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict
WARNING: Skipping page https://pypi.org/simple/pip/ because the GET request got Content-Type: .The only supported Content-Type is text/html
ERROR: Cannot install pip==20.0.2 because these package versions have conflicting dependencies.
ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/user_guide/#fixing-conflicting-dependencies

Copy link

@rth rth left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks a lot for doing this work! It's certainly more work that I anticipated (without knowing much about cibuildwheel) :)

cibuildwheel/resources/pyodide_python.sh Outdated Show resolved Hide resolved
docs/options.md Outdated Show resolved Hide resolved
@henryiii
Copy link
Contributor

@henryiii is the windows_x86 failure a flake?

Yes. Some networking issue that is hard to track down.

@henryiii
Copy link
Contributor

henryiii commented May 1, 2023

BTW, would you like me to rebase for you? I don't mind.

@hoodmane
Copy link
Contributor Author

hoodmane commented May 1, 2023

I already merged main locally.

@hoodmane
Copy link
Contributor Author

hoodmane commented May 6, 2023

Okay I think this is working now. Would appreciate a review @henryiii, @joerick

@hoodmane
Copy link
Contributor Author

hoodmane commented May 6, 2023

Presumably we need documentation somewhere for the fact that Python 311 needs to be installed.

docs/options.md Outdated Show resolved Hide resolved
.pre-commit-config.yaml Outdated Show resolved Hide resolved
.github/workflows/test.yml Outdated Show resolved Hide resolved
.github/workflows/test.yml Outdated Show resolved Hide resolved
.github/workflows/test.yml Outdated Show resolved Hide resolved
cibuildwheel/__main__.py Outdated Show resolved Hide resolved
cibuildwheel/pyodide.py Outdated Show resolved Hide resolved
cibuildwheel/pyodide.py Outdated Show resolved Hide resolved
cibuildwheel/pyodide.py Outdated Show resolved Hide resolved
cibuildwheel/pyodide.py Outdated Show resolved Hide resolved
test/test_emscripten.py Outdated Show resolved Hide resolved
@joerick
Copy link
Contributor

joerick commented May 12, 2023

Splendid work on this so far @hoodmane ! Quality looks really good. Comments above.

@henryiii
Copy link
Contributor

We talked about the Python requirement at PyCon a bit - I think it's best to require 3.11 instead of adding a lot of time & bandwidth for a docker image. Since pyodide is a bit behind official CPython, all CI's should have a suitable host Python available. If PyBI's land, then we might be able to make this simpler. :)

I think the name is a bit in the air. Technically, it's emscripten + version + flags. But pyodide selects a specific set, and the hope is it won't update between Python versions. So 3.12 would be a new version of emscripten + flag updates, etc. But I think that's going to be incorrect at least once (I think there's a planned 3.11 update). The hope is by the time wheels are cleared for PyPI, it will be stable. There's also the possibility that CPython will move emscripten to tier 2 (not soon, sadly, looks like only WASI is going to tier 2), in which case CPython could take over the definition of the emscripten version + flags for whatever version it started on.

This will need docs - it should be clear this is experimental and you can't upload the wheels to PyPI yet. Setting it up is a bit more involved than just adding it to a matrix, etc. I can help with docs later (traveling at conference this week).

Personally and quite selfishly, I really want the cmake bug fixed before it ships, I expect scikit-build/scikit-build-core users will want to make pyodide wheels once this ships in cibuildwheel, and can't do that due to a variable being stripped from CMake's environment by emscripten and pyodide isn't auto-fixing out-of-tree wheels. I can help fix this early next week.

@hoodmane
Copy link
Contributor Author

Thanks for the detailed review @joerick!

@hoodmane
Copy link
Contributor Author

pyodide isn't auto-fixing out-of-tree wheels

You're talking about replace_so_abi_tags()? I think we can release a Pyodide 0.23.3 that calls that for out of tree wheels.

@ryanking13 do you see any reason not to call replace_so_abi_tags() when building out of tree? This probably fits into your project of making the out of tree builds more the same as the in-tree ones.

Of course it would be great if we could also fix the issue with Emscripten, but it seems potentially easier to change Pyodide.
emscripten-core/emscripten#19301

@ryanking13
Copy link

@ryanking13 do you see any reason not to call replace_so_abi_tags() when building out of tree? This probably fits into your project of making the out of tree builds more the same as the in-tree ones.

Sounds good. We should try to make out of tree and in tree builds identical.

@henryiii
Copy link
Contributor

You're talking about replace_so_abi_tags()?

There are three fixes, any one of them would be enough:

  1. Run replace_so_abi_tags() for out-of-tree builds. It's a bit of a hack, but it's a minimum to get things working.
  2. Replace emcmake with pyodide's own implementation. This has to be pretty easy, it's a really simple script. This would allow scikit-build-core to get the extension right to begin with, and would support FindPython properly I think.
  3. Fix emscripten to not strip this variable when using emcmake. That's a harder fix, and would require an upstream change.

I'd really like both 1 and 2 if possible. :)

@hoodmane
Copy link
Contributor Author

hoodmane commented May 15, 2023

Okay I think all the integration tests are passing and I think I addressed the most recent round of review comments. So this is waiting on:

  • review round 2?
  • fix the so_abi problems

@henryiii I'll make a PR to run replace_so_abi_tags for out of tree builds, you can work on an emcmake replacement. Then we can release pyodide 0.23.3 and update cibuildwheel to use it. Then we can merge this (assuming that I address any further review comments).

.github/workflows/test.yml Outdated Show resolved Hide resolved
.github/workflows/test.yml Outdated Show resolved Hide resolved
test/conftest.py Outdated Show resolved Hide resolved
test/test_abi_variants.py Outdated Show resolved Hide resolved
test/test_abi_variants.py Outdated Show resolved Hide resolved
test/test_before_test.py Outdated Show resolved Hide resolved
test/test_custom_repair_wheel.py Outdated Show resolved Hide resolved
test/test_custom_repair_wheel.py Outdated Show resolved Hide resolved
cibuildwheel/resources/build-platforms.toml Outdated Show resolved Hide resolved
test/test_testing.py Outdated Show resolved Hide resolved
@hoodmane
Copy link
Contributor Author

I guess it's failing because of the color / formatting in pyodide build -h.

@hoodmane
Copy link
Contributor Author

Hmm test passes locally...

@@ -19,12 +19,11 @@ def test_build_frontend_args(tmp_path, capfd, frontend_name):
project.generate(project_dir)

# the build will fail because the frontend is called with '-h' - it prints the help message
add_env = {"CIBW_BUILD_FRONTEND": f"{frontend_name}; args: -h"}
if utils.platform == "pyodide":
add_env["TERM"] = "dumb" # disable color / style
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hopefully this works? I have no idea why it passes locally though??

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems that it fixed it indeed =)

Copy link
Member

@mayeut mayeut May 27, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it seems too have been style/color as you pointed out.
I don't know why it passes locally either...

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe something is checking for the GitHub actions var and assuming it is in a CI run that supports color?

Copy link
Contributor

@henryiii henryiii left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMO, this is ready to merge, we can do followup patches if we find any improvements. It looks like the two outstanding issues might both be on the pyodide side, so we might not want to make a release until 0.26.1.

Unless anyone has any reservations, will merge tomorrow.

@henryiii
Copy link
Contributor

Edit: I guess the manual one was on Linux,

Yes, scikit-hep/boost-histogram#938 (linux) makes it through the "pytest" command and dies on the other bug. But we were seeing this issue in cibuildwheel's tests, even on Linux.

@henryiii henryiii merged commit 9f2a3cb into pypa:main May 28, 2024
24 checks passed
@henryiii
Copy link
Contributor

Thanks everyone!

@hoodmane hoodmane deleted the emscripten branch May 28, 2024 12:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Support for wasm32-emscripten wheels Pyodide Wheels
10 participants